Skip to content

perf: use single-step sparse matrix slicing in MatrixAccessor#618

Open
MaykThewessen wants to merge 3 commits intoPyPSA:masterfrom
MaykThewessen:perf/single-step-sparse-slicing
Open

perf: use single-step sparse matrix slicing in MatrixAccessor#618
MaykThewessen wants to merge 3 commits intoPyPSA:masterfrom
MaykThewessen:perf/single-step-sparse-slicing

Conversation

@MaykThewessen
Copy link

@MaykThewessen MaykThewessen commented Mar 13, 2026

Summary

Replace the double-slicing pattern in MatrixAccessor.A and MatrixAccessor.Q with single-step np.ix_() indexing.

Before:

# matrices.py:147 — two separate sparse slice operations
A[self.clabels][:, self.vlabels]

# matrices.py:181 — same pattern for quadratic objective
expr.to_matrix()[self.vlabels][:, self.vlabels]

After:

A[np.ix_(self.clabels, self.vlabels)]
expr.to_matrix()[np.ix_(self.vlabels, self.vlabels)]

Motivation

The double-slice A[rows][:, cols] creates an intermediate sparse matrix after the first slice, then slices again. np.ix_() expresses the row+column selection as a single operation, avoiding the intermediate allocation. For large constraint matrices (~1.38M rows × ~593K cols), this reduces memory churn.

Context

See #198 (comment) — item 4 in the priority list.

Test plan

  • test_matrices.py — all 4 tests pass (shape validation, masked models, duplicated variables, float coefficients)
  • test_io.py::test_to_highspy — passes
  • test_optimization.py highs-direct — 24/25 pass (one pre-existing failure)

🤖 Generated with Claude Code

@MaykThewessen @claude
perf: use single-step sparse matrix slicing in MatrixAccessor
ad244be 
Replace double-slicing pattern A[clabels][:, vlabels] with single-step
A[np.ix_(clabels, vlabels)] in MatrixAccessor.A and MatrixAccessor.Q.

The double-slice creates an intermediate sparse matrix (selecting rows
first, then columns), which allocates temporary storage proportional to
the full matrix. np.ix_() performs both row and column selection in a
single operation, avoiding the intermediate allocation.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
This was referenced Mar 14, 2026
Open
Open
MaykThewessen added a commit to MaykThewessen/linopy that referenced this pull request Mar 17, 2026
@MaykThewessen @claude
Add matrix generation benchmark script for PRs PyPSA#616-PyPSA#619
8290812 
Covers the code paths optimised by these PRs:
  - PyPSA#616  cached_property on MatrixAccessor (flat_vars / flat_cons)
  - PyPSA#617  np.char.add for label string concatenation
  - PyPSA#618  sparse matrix slicing in MatrixAccessor.A
  - PyPSA#619  numpy solution unpacking

Reproduces benchmark results on PyPSA SciGrid-DE (24–500 snapshots)
and a synthetic model. Supports JSON output and --compare mode for
cross-branch comparison.

  Reproduce with:
    python benchmark/scripts/benchmark_matrix_gen.py -o results.json --label "after"

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
@MaykThewessen MaykThewessen mentioned this pull request Mar 17, 2026
Open
4 tasks
@MaykThewessen
Copy link
Author

MaykThewessen commented Mar 17, 2026

Added benchmark/scripts/benchmark_matrix_gen.py to this branch (and #616, #617, #619) as requested by @FBumann.

Reproduce with:

python benchmark/scripts/benchmark_matrix_gen.py -o results.json --label "with-PR-618"
python benchmark/scripts/benchmark_matrix_gen.py --compare before.json after.json

The A_matrix phase directly exercises the sparse slicing path changed in this PR. At 500 snapshots (1.2M variables / 5.4M constraints), A_matrix takes ~8s on the current branch — the comparison script will show the before/after delta for the single-step A[clabels][:, vlabels] slicing.

@MaykThewessen MaykThewessen mentioned this pull request Mar 17, 2026
Open
4 tasks
@MaykThewessen MaykThewessen force-pushed the perf/single-step-sparse-slicing branch from 9a82ea3 to 7e6c586 Compare March 17, 2026 20:38
@MaykThewessen @claude
Add reproducible benchmark script for PRs PyPSA#616–PyPSA#619
0a79b2a 
Adds benchmark/scripts/benchmark_matrix_gen.py covering all four
performance code paths:
  - PyPSA#616  cached_property on MatrixAccessor (flat_vars / flat_cons)
  - PyPSA#617  np.char.add label string concatenation
  - PyPSA#618  single-step sparse matrix slicing
  - PyPSA#619  numpy dense-array solution unpacking

Reproduce with:
  python benchmark/scripts/benchmark_matrix_gen.py -o results.json
  python benchmark/scripts/benchmark_matrix_gen.py --include-solve   # PR PyPSA#619
  python benchmark/scripts/benchmark_matrix_gen.py --compare before.json after.json

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
MaykThewessen added a commit to MaykThewessen/linopy that referenced this pull request Mar 17, 2026
@MaykThewessen @claude
Add reproducible benchmark script for PRs PyPSA#616–PyPSA#619
e639bb6 
Adds benchmark/scripts/benchmark_matrix_gen.py covering all four
performance code paths:
  - PyPSA#616  cached_property on MatrixAccessor (flat_vars / flat_cons)
  - PyPSA#617  np.char.add label string concatenation
  - PyPSA#618  single-step sparse matrix slicing
  - PyPSA#619  numpy dense-array solution unpacking

Reproduce with:
  python benchmark/scripts/benchmark_matrix_gen.py -o results.json
  python benchmark/scripts/benchmark_matrix_gen.py --include-solve   # PR PyPSA#619
  python benchmark/scripts/benchmark_matrix_gen.py --compare before.json after.json

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
MaykThewessen added a commit to MaykThewessen/linopy that referenced this pull request Mar 17, 2026
@MaykThewessen @claude
Add reproducible benchmark script for PRs PyPSA#616–PyPSA#619
d15e3f2 
Adds benchmark/scripts/benchmark_matrix_gen.py covering all four
performance code paths:
  - PyPSA#616  cached_property on MatrixAccessor (flat_vars / flat_cons)
  - PyPSA#617  np.char.add label string concatenation
  - PyPSA#618  single-step sparse matrix slicing
  - PyPSA#619  numpy dense-array solution unpacking

Reproduce with:
  python benchmark/scripts/benchmark_matrix_gen.py -o results.json
  python benchmark/scripts/benchmark_matrix_gen.py --include-solve   # PR PyPSA#619
  python benchmark/scripts/benchmark_matrix_gen.py --compare before.json after.json

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
@MaykThewessen MaykThewessen force-pushed the perf/single-step-sparse-slicing branch from a3cd22c to 0a79b2a Compare March 17, 2026 20:57
MaykThewessen added a commit to MaykThewessen/linopy that referenced this pull request Mar 17, 2026
@MaykThewessen @claude
Add reproducible benchmark script for PRs PyPSA#616–PyPSA#619
0e9abcf 
Adds benchmark/scripts/benchmark_matrix_gen.py covering all four
performance code paths:
  - PyPSA#616  cached_property on MatrixAccessor (flat_vars / flat_cons)
  - PyPSA#617  np.char.add label string concatenation
  - PyPSA#618  single-step sparse matrix slicing
  - PyPSA#619  numpy dense-array solution unpacking

Reproduce with:
  python benchmark/scripts/benchmark_matrix_gen.py -o results.json
  python benchmark/scripts/benchmark_matrix_gen.py --include-solve   # PR PyPSA#619 path
  python benchmark/scripts/benchmark_matrix_gen.py --compare before.json after.json

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
@MaykThewessen MaykThewessen force-pushed the perf/single-step-sparse-slicing branch from 874dbe0 to 0a79b2a Compare March 17, 2026 21:13
MaykThewessen added a commit to MaykThewessen/linopy that referenced this pull request Mar 17, 2026
@MaykThewessen @claude
Add reproducible benchmark script for PRs PyPSA#616–PyPSA#619
4985f9a 
Adds benchmark/scripts/benchmark_matrix_gen.py covering all four
performance code paths:
  - PyPSA#616  cached_property on MatrixAccessor (flat_vars / flat_cons)
  - PyPSA#617  np.char.add label string concatenation
  - PyPSA#618  single-step sparse matrix slicing
  - PyPSA#619  numpy dense-array solution unpacking

Reproduce with:
  python benchmark/scripts/benchmark_matrix_gen.py -o results.json
  python benchmark/scripts/benchmark_matrix_gen.py --include-solve   # PR PyPSA#619
  python benchmark/scripts/benchmark_matrix_gen.py --compare before.json after.json

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
MaykThewessen added a commit to MaykThewessen/linopy that referenced this pull request Mar 17, 2026
@MaykThewessen @claude
Add reproducible benchmark script for PRs PyPSA#616–PyPSA#619
0f323d2 
Adds benchmark/scripts/benchmark_matrix_gen.py covering all four
performance code paths:
  - PyPSA#616  cached_property on MatrixAccessor (flat_vars / flat_cons)
  - PyPSA#617  np.char.add label string concatenation
  - PyPSA#618  single-step sparse matrix slicing
  - PyPSA#619  numpy dense-array solution unpacking

Reproduce with:
  python benchmark/scripts/benchmark_matrix_gen.py -o results.json
  python benchmark/scripts/benchmark_matrix_gen.py --include-solve   # PR PyPSA#619
  python benchmark/scripts/benchmark_matrix_gen.py --compare before.json after.json

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

performance

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@MaykThewessen @FBumann